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Abstract 

Anomalous couplings of the Higgs boson are searched for through the processes 
e"'"e~ —>■ H7, e"'"e~ e"'"e~H and e"'"e~ —>■ HZ. The mass range 70 GeV < mn < 
190 GeV is explored using 602 pb~^ of integrated luminosity collected with the L3 
detector at LEP at centre-of-mass energies a/s = 189 — 209 GeV. The Higgs decay 
channels H ^ ff, H ^ 77, H ^ Z7 and H — * WW^*^ are considered and no 
evidence is found for anomalous Higgs production or decay. Limits on the anomalous 
couplings d, (Ib, Agf^, Ak^ and are derived as well as limits on the H — > 77 and 
H — i> Z7 decay rates. 



Submitted to Phys. Lett. B 



1 Introduction 



The mechanism of spontaneous symmetry breaking is a cornerstone of the Standard Model of 
the electroweak interactions [1]. It explains the observed masses of the elementary particles 
and postulates an additional particle, the Higgs boson. Despite its relevance, experimental 
information on the Higgs boson is scarce and indirect. It leaves room for deviations from the 
Standard Model expectations such as anomalous couplings of the Higgs boson. 

The Standard Model can be extended, via a linear representation of the SU{2)l x U{1)y 
symmetry breaking mechanism [2] , to higher orders where new interactions between the Higgs 
boson and gauge bosons become possible. These modify the production mechanisms and decay 
properties of the Higgs boson. The relevant CP- invariant Lagrangian terms are [3] : 

+ g'^^^ Z^JL^d^^ + g^^^^ WL^JL^^ + gf^^ HZ^Z^ 
+ ^iw (W+ W^ia'^H + h.c) + g^S^^ HW+ Wr, (1) 

where A^, Z^, and H are the photon, Z, W and Higgs fields, respectively, and X^,^ = 
— di^^j,. The couplings in this Lagrangian are parametrized as [4-6]: 

5'H77 = 77-^^ — ((isin^6'H/ + (ifi cos^^^vi^) (2) 

S'hzt = (AgTi sin26liy - Ak^ tan6'wA) (3) 

777.W 

9bL^ = ^ sm2^iy {d-dB) (4) 
9'hzz = (A^i cos26lvF + A«;^tan^6'vi^) (5) 

dSzz = 7r~ {dcos^6w + dBsin^ew) (6) 
(3) _ gmw . 

(1) _ 9 . 7 /ON 
5'hww - ^2 ^^1 v^J 

5'hww ^ 7^ ' (^) 

mw cos2&v^ 

where g is the SU{2)l coupling constant, Oyy is the weak mixing angle and m\v and rriz represent 
the masses of the W and Z bosons, respectively. The five dimensionless parameters d, ds, ^gf, 
Ak^ and 6z constitute a convenient set to describe deviations in the interactions between the 
Higgs boson and gauge bosons. They are not severely constrained by electroweak measurements 
at the Z pole or at lower energies [3,7]. 

The couplings d and ds were introduced in Reference 4, while Agf and Ak,^ also describe 
possible deviations in the couplings of W bosons with photons and Z bosons [5]. A search 
for anomalous Higgs production and decay with non-vanishing values of Agf or Ak^ is a 
complementary study to the analysis of triple-gauge-boson couplings in the e'^e" — > W"'"W~ 
process. The parameter = {1 + SzY describes a global rescaling of all Higgs couplings and 
affects the Higgs production cross section, but not its branching fractions [6]. 

We search for a Higgs particle produced in the e+e" — > H7 and c+e~ e"''e^H processes 
shown in Figures la and lb. Their rates would be enhanced in presence of anomalous H77 and 
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HZ7 couplings. These processes probe Higgs masses, mn, up to the centre-of-mass energy of 
the colhsion, ^/s. For mn < — rriz-, this analysis is complemented by the results from the 
L3 searches for the e+e~ — > HZ process [8,9], which are sensitive to anomalous HZZ and HZ7 
couplings, as shown in Figure Ic. 

The existence of H77 and HZ7 couplings would lead to large H — > 77 and H ^ Z7 branching 
fractions, which at tree level are zero in the Standard Model. These decay modes have comple- 
mentary sensitivities and allow to probe a large part of the parameter space. In addition, the 
decay H WW*^*-* would also be enhanced in the presence of anomalous HWW couplings. 

The data used in this analysis were collected with the L3 detector [10] at LEP at \/i = 189 — 
209 GeV and correspond to an integrated luminosity of 602 pb~^. Searches for anomalous Higgs 
production were previously performed, with data of lower energy and integrated luminosity, 
by L3 and other experiments [11,12]. Other non-standard Higgs searches performed at LEP 
are reported in [9, 13]. The results reported in this Letter include and supersede those of 
Reference 11. 

2 Analysis strategy 

Table 1 summarizes the experimental signatures considered for the study of Higgs anomalous 
couplings according to the different production mechanisms and decay channels. 

For the e"'"e~ — > H7 process, the decay channels H — > 77, H ^ Z7 and H WW*^*^ are 
investigated. Only hadronic decays of Z and W bosons are considered. 

For the e"'"e~ e"'"e~H process, only the H — > 77 decay is studied. The H ^ bb decay 
was considered in the study of the e"'"e~ — > e+e~H and e''"e~ — > H7 processes for data collected 
at ^/s = 189 GcV [11]. This decay is dominant for ttih ^ mz, where H — > 77 is strongly 
suppressed and H ^ Z7 is kinematically forbidden. At the centre-of-mass energies considered 
in this Letter, this region is efficiently covered by an interpretation of the results of the search 
for the e+e^ — > HZ process [8] and the H — > bb decay is not considered here. 

No dedicated selection is devised for the e+e~ — > HZ process and the limits obtained by L3 
in the searches for the Standard Model Higgs boson and for a fermiophobic Higgs boson are 
interpreted in terms of anomalous Higgs couplings. 

The analysis is performed as a function of mn in steps of 1 GcV. The H 77, H ^ Z7 
and H WW^*) decays probe the ranges 70 GeV < mn < 190 GeV, 95 GeV < mn < 190 GeV 
and 130 GeV < mn < 190 GeV, respectively. 

After the event selections described below, variables which depend on mn are built to 
discriminate signal and background. Finally, the number of events in a mass window around 
the mn value under study is compared with the Standard Model expectation and interpreted 
in terms of cross sections and anomalous couplings. 

3 Data and Monte Carlo samples 

Table 2 hsts the centre-of-mass energies and the corresponding integrated luminosities used in 
this analysis. The data at ^/s — 189 GeV are re-analysed for the e+e" —>■ H7 — > 777 and 
e"'"e~ — s> e+e^H — > e"'"e~77 channels and results for the full range ^/s = 189 — 209 GeV are 
reported here. All other analyses discussed in this Letter refer to the y/s = 192 — 209 GeV 
range, and their results are then combined with those obtained at = 189 GeV [11]. 
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To describe the e"'"e~ — > H7 process we wrote a Monte Carlo generator which assumes a 
1 + cos^ dependence of the differential cross section as a function of the cosine of the Higgs 
production angle, ^^h- It includes effects of initial-state [14] and final-state [15] radiation as well 
as spin correlations and off- shell contributions in cascade decays such as H — > Z7 — > ff7. 

The e+e~ — > e'*"e~H process is interpreted as the production of a narrow- width spin-zero 
resonance in two-photon collisions, and modelled with the PC Monte Carlo generator [16]. 

The differential cross section of the process e+e~ — * HZ in the presence of anomalous 
couplings is taken from Reference 17. References 18 and 19 are used for the branching fractions 
and partial widths of a Higgs boson with anomalous couplings. The interference between the 
e+e~ — > HZ process in the Standard Model and in presence of anomalous couphngs [17] is taken 
into account in the simulation. It is negligible for the e+e" — > H7 and e+e" — > e+e^H cases. 

Signal events are generated for 70 GeV < mn < 190 GeV, in steps of 20 GeV. More than 
5000 signal events are generated for each value of mn and for each process under study. For 
intermediate values of the Higgs mass, the signal efficiency is interpolated between the generated 
values. 

Standard Model processes are modelled with the following Monte Carlo generators: GGG [20] 
for e+e" 77(7), KK2f [21] for e+e" qq(7), PYTHIA [22] for e+e" ^ ZZ and e+e" ^ 
Ze+e", KORALW [23] for e+e" ^ W+W-(7) and EXCALIBUR [24] for e+e" ^ Weu and 
other four-fermion final states. 

The L3 detector response is simulated using the GEANT program [25] which takes into ac- 
count effects of energy loss, multiple scattering and showering in the detector. Time-dependent 
detector inefficiencies, as monitored during the data-taking period, are included in the simula- 
tions. 

4 Event selection 

All analyses presented in this Letter rely on photon identification. Photon candidates are 
defined as clusters in the electromagnetic calorimeter with a shower profile consistent with 
that of a photon and no associated track in the tracking chamber. To reduce contributions 
from initial-state and final-state radiation, photon candidates must satisfy Ej > 5 GeV and 
I cos^-y |< 0.97, where is the photon energy and 6^ its polar angle. 

Events with hadronic decays of the Z and W bosons in the H ^ Z7 and H ^ WW^*) 
channels are pre-selected requiring high particle multiplicity and a visible energy, E'^s, satisfying 
0.8 < E^is/^ < 1.2. 

4.1 The e+e" — > H7 — > 777 analysis 

Events from the e+e~ — > H7 — > 777 process are selected by requiring three photon candidates in 
the central region of the detector, | cos^^| < 0.8, with a total electromagnetic energy larger than 
a/s/2. Out of the three possible two-photon combinations, the one with a mass, m^^, closest 
to the mn hypothesis under investigation is retained. As an example. Figure 2a presents 
the distribution of m^^ for mn = 110 GeV. The event is accepted as a Higgs candidate if 
I m-y^ — mn |< 0.05 mn. 

The numbers of events observed and expected in the full data sample at y/s — 189—209 GeV 
are shown in Table 3 for several mn hypotheses. The contamination from processes other than 
e"'"e~ — 77(7), as estimated from Monte Carlo simulations, is found to be negligible. The 
signal selection efficiency is in the range 25% — 30%, depending on mn and ^/s. 
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4.2 The e"'"e — > e'^e H — > e'^e 77 analysis 

In the process e'^e" — > e+e~H, the final state e~ and e+ tend to escape detection at low polar 
angles, originating events with missing longitudinal momentum and missing mass. The selection 
requires two photon candidates from the H — > 77 decay in the central region of the detector. 
A kinematic fit is performed assuming the missing momentum to point in the beam pipe and 
the visible mass of the event to be consistent, within the experimental resolution, with the mn 
hypothesis under investigation. The distribution of the of the fit is shown in Figure 2b for 
mn = 130 GeV. Events are accepted as Higgs candidates if < 50 — 0.2 GeV~^ x mn. The 
dependence of the cut on mn refiects the decrease of the background contribution for increasing 
values of mu- 

The numbers of events observed and expected in the full data sample at \/s = 189—209 GeV 
are shown in Table 3 for several mu hypotheses. The background comes from e"''e~ — > 77(7) 
events. The signal selection efficiency varies from 20% to 30%, with a smooth dependence on 
mn and y/s. 

4.3 The e+e" H7 Z77 analysis 

Pre-selected hadronic events with two isolated high energy photons are considered for the 
e"'"e~ H7 — Z77 analysis. Events are retained which have a recoiling mass, m^ec, calculated 
from the four- momenta of the two photons, compatible with rriz'- 80 GeV < m^ec < 110 GeV. 
The hadronic system is clustered into two jets with the DURHAM [26] algorithm and a kine- 
matic fit, in which the jet angles are fixed and the jet energies can vary, is performed to improve 
the resolution on the reconstructed Z-boson mass. Of the two possible combinations of two jets 
and a photon, the one is retained with mass, rUq^j, closer to the mn hypothesis under investiga- 
tion. The distribution of m^q^ is shown in Figure 2c for mn = 150 GeV. An event is considered 
as a Higgs candidate if |mqq^ — mnl < 15 GeV. 

The numbers of events observed and expected in the data sample at ^/s — 192 — 209 GeV 
are shown in Table 3 for several mn hypotheses. The signal selection efficiency is around 22%. 
The background is dominated by resonant e"'"e~ — > Z77 production (70%) with contributions 
from the e+e" — > qq(7) process and four-fermion final states. 

4.4 The e+e- ^ H7 ^ WW(*)7 analysis 

The energy of the photon in the e+e" — > H7 WW*-*-'7 process depends on mn as El^^{mY{) = 
[s — 171^)1 Pre-selected hadronic events are retained if they have a photon with energy 
compatible with the mn hypothesis under investigation, El^^^{mYi + 2Q GeV) < < E!^^'^{mH_ — 
20 GeV). If multiple photon candidates are observed, the photon is retained which has an energy 
closest to £^^^'^(mH). The rest of the event is clustered into four jets by means of the DURHAM 
algorithm. 

A kinematic fit, in which the jet angles arc fixed and the jet energies can vary, is performed 
to improve the resolution on the reconstructed W-boson mass. For mn > 2mw both W bosons 
are on-shell and the constraint that both invariant jet-jet masses be compatible with mw is 
included in the fit. For mn < 2mw one of the W bosons is off-shell and only one of the invariant 
jet-jet masses is required to be compatible with mw- The fit is repeated for all possible jet 
pairings and the pairing is chosen for which the of the fit is minimal. An event is considered 
as a Higgs candidate if < 6.0 for the hypothesis mn < 2mw or x^ < 15.0 for mn > 2mw 
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The invariant mass of the four-jet system, mqqqq, estimates mn- Its distribution is presented 
in Figure 2d for mn = 170 GeV. 

The numbers of events observed and expected in the data sample at = 192 — 209 GeV 
are shown in Table 3 for several mn hypotheses. The signal selection efficiency is around 25%, 
for 150 GeV < mn < 170 GeV, decreasing to about 20% for masses out of this range. A small 
dependence on ^/s is observed. The background is dominated by the processes e'^e"' — > qq(7) 
and e+e~ — > W+W~(7), which is above 65% for mn > 150 GeV. 



5 Cross sections limits 

The results of all the analyses agree with the Standard Model predictions and show no evidence 
for a Higgs boson with anomalous couplings in the mn mass range under study. Upper limits on 
the product of the production cross sections and the corresponding decay branching fractions 
are derived [27] at the 95% confidence level (CL). The cross section of the e"'"e~ e"'"e~H 
process is proportional to the partial Higgs width into photons, r(H 77), and limits are 
quoted on r(H — >• 77) x Br(H — >• 77). 

In order to combine data sets at different ^/s values, a dependence of the type a'^^{^) = 
C,a^^{-y/s) is assumed for the cross section of anomalous Higgs production, a"^^ . The e+e~ 
H7 production cross section in the Standard Model, , accounts for the dominant dependence 
on ^/s while C is a parameter which does not depend on \/s. Limits on C, are derived and 
interpreted as cross section limits at the luminosity-averaged centre-of-mass energy < ^/s >— 
197.8 GeV. 

The cross section hmits for the investigated processes are given in Figure 3 together with 
the expectations for non-zero values of the anomalous couplings. 



6 Limits on anomalous couplings 

6.1 Results from e"'"e~ — > HZ with H — >^ ff or H — > 77 

The process e"'"e~ — > HZ, with H — > ff , studied in Reference 8, is sensitive to anomalous HZZ 
and HZ7 couplings in the Higgs production vertex. In addition, the process e"'"e~ — > HZ with 
H — > 77, object of the search for a fermiophobic Higgs [9], is sensitive to the H77 coupling in 
the decay vertex. 

Limits on the coupling are derived from the results of our search for the Standard Model 
Higgs boson [8] . They are obtained by interpreting as a scale factor of the Higgs production 
cross section and are shown in Figure 4. They include the systematic uncertainties on the 
search for the Standard Model Higgs boson [8]. 

The limits on the couplings d, ds, i^g\ and Ak-j, are extracted from the numbers of observed 
events, expected background and signal events reported in References 8 and 9. These limits are 
driven by the size of the deviations of the product cr^^ x Br^^ with respect to x Br'^^, 
where Br"^*" and Br'^^ denote the Higgs branching ratios in the presence of anomalous couplings 
and in the Standard Model respectively. The ratios R = {a^'-' x Br"^^) / (a^^ x Br*^^) are shown 
in Figure 5 for H — > f f and H — > 77, for mn = 100 GeV. 

The H — > ff and H — ^ 77 channels have different behaviours with respect to the parameters 
d and ds, as these describe the H77 couphng. The parameters Agf and Akj describe the 
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HZ7 and HZZ couplings and hence affect only the Higgs production vertex in the e'^e — > HZ 
process. They give similar deviations for both the H — > ff and H — > 77 channels. 

6.2 One-dimensional limits 

Figure 6 presents the limits on d, ds, Ag^f and A function of mn- A couphng at the 

time is considered, fixing the others to zero. Limits from the most sensitive channels are shown 
in addition to the combined results. 

The region mn < y/s — niz is excluded by the e+e~ — > HZ search for any value of the four 
couphngs. The fermiophobic search e"^e~ WL, with H — > 77, is sensitive to large values of d 
and ds- for which there is an enhancement of the H — > 77 branching fraction. The standard 
search e+e^ HZ, with H ^ bb or r+r^, covers the region d ds ^ A region for 
mn ~ 97 GeV in the d vs. mn plane of Figure 6a is not excluded due to an excess of events 
observed in the e^e^ — > HZ search [28]. 

The e+e~ — > H7 — > 777 and e+e~ — > e+e~H — > e+e~77 channels have a large sensitivity 
if the H77 coupling is large, i.e. when dsin^^^ + dBCOS^Ow has a sizable value (Figures 6a 
and 6b). On the other hand, the e"'"e~ — > H7 Z77 process has a dominant role when the 
channel H ^ 77 is suppressed, which occurs for the couplings and A/t-y in the mass region 

< < 2mw (Figures 6c and 6d). 

The contribution from the e'^e" — > H7 — > WW(*)7 process to the hmits presented in Figures 
6a and 6c is small and restricted to mn ~ 160 GeV. This happens since a large decay width 
for H WW'-*^ corresponds to large values of d or which also imply large widths for the 
competing modes H — > 77 and H Z7. 

The sensitivity of the analysis degrades rapidly when mn approaches the 2mw threshold, 
where the H — > 77 and H — > Z7 are no longer dominant, even in the presence of relatively large 
anomalous couplings. 

Several sources of systematic uncertainties are investigated and their impact on the signal 
efficiency and background level is evaluated. The limited Monte Carlo statistics affects the sig- 
nal by less than 2% and the background by 8% for the photonic channels and less than 4% for 
the hadronic channels. The accuracy of the cross section calculation for background processes 
adds less than 0.4% to the uncertainty in the background normalisation. The systematic un- 
certainty due to the selection procedure was estimated by varying the most important selection 
criteria and was found to be less than 1%. In particular, the effect of the limited knowledge of 
the energy scale of the electromagnetic calorimeter has a small impact in the limits. 

The combined effect of the systematic uncertainties is included in the limits shown in Fig- 
ure 6. It degrades the limits by at most 4%, shghtly depending on the couphng and the Higgs 
mass hypothesis. 

We verified that possible effects of angular dependence of the efficiency on the value of the 
anomalous couplings is negligible for the e+e~ — > HZ process. No such effects are expected for 
the e"*'e" — > e"*'e~H and e+e" — > H7 processes. 

6.3 Two-dimensional limits 

Assuming the absence of large anomalous WWZ and WW7 couplings, i.e. lS.g\ = Ak,^ = [29], 
the H77 and HZ7 couplings are parametrized via the following subset of effective operators: 

>Ceff = gu-y-y HA^.A'^^ + g'^ RA^^Z^"" + h.c. (10) 
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where the dependence of gu-yy and g^l^ on the d and ds couphngs is given by Equations 2 
and 4. This Lagrangian is used to compute the maximal partial widths and branching fractions 
of the decays H ^ Z7 and H 77, allowed by the limits on d and ds- The results are 
presented in Figure 7 for two different Higgs masses, in the region of interest for Higgs searches 
at future colliders. The results are consistent with the tree level Standard Model expectations 

r(H ^ Z7) ^ r(H ^ 77) ^ 0. 
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Production 
mechanism 


Decay mode 


H — > 77 


Z7 


H ^ WW(*) 


H^ff 


e'^e" — > H7 
e+e~ He+e" 
e+e- ^ HZ 


37 
27 +^ 
2 7 + ff [9] 


27 + 2 jets 


1 7 + 4 jets 


1 7 + bb [11] 
bb + ^ [11] 
fff'P [8] 



Table 1: Experimental signatures for the search for anomalous couplings in the Higgs sector. 
The symbol denotes missing energy and momentum. Searches in the e'*'e~ H7 hh'y and 
e+e~ e+e~H — > e+e~"bb channels are only performed at a/s — 189 GeV [11]. 



(GeV) 
£ (pb-i) 


188.6 


191.6 195.5 


199.5 


201.7 


204.8 206.6 


176.8 


28.8 82.4 


67.6 


36.1 


74.7 135.6 



Table 2: Average centre-of-mass energy and integrated luminosity of the data samples used for 
the search for anomalous couplings in the Higgs sector. 
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74.7 26.8 


190 
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8.9 


22.9 


24 


29.5 


30.1 


96 


101.0 


22.5 


113 


107.3 19.5 



Table 3: Numbers of observed, Nd, and expected, A''^, events and signal selection efficiencies, 
e, for different analysis channels and values of the Higgs mass. Centre-of-mass energies in 
the range 189 GeV < ^/s < 209 GeV are considered for the e'^e" — > H7 — > 777 and e'*"e"" — > 
e+e~H — > e"'"e~77 channels, while the e+e~ H7 — > Z77 and e"'"e~ — > H7 — > WW^*^7 channels 
are analysed in the 192 GeV < ^/s < 209 GeV range. 
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Figure 2: Distributions of the final discriminant variables for a) the e"'"e~ — > 777 channel: the 
mass, m-y-y, of the two-photon system; b) the e"'"e~ — > e''"e~77 channel: the of the constrained 
fit; c) the e^e~ — > Z77 channel: the mass, mqq-y, of the system of the two-jets and a photon 
and d) the c"'"c~ WW*-*^7 channel: the mass, mqqqq, of the hadronic system. The points 
represent the data, the open histograms the background and the hatched histograms the Higgs 
signal with an arbitrary cross section of 0.1 pb. The Higgs mass hypotheses indicated in the 
figures are considered. The arrows indicate the values of the cuts. 
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Figure 3: Upper limits at 95% CL as a function of the Higgs mass on: a) a{e'^e — > H7) x 
Br(H ^ 77); b) r(H ^ 77) X Br(H ^ 77); c) a{e+e- H7) x Br(H ^ Z7); d) a(e+e- ^ 
H7) X Br(H — » WW^*-*). The dashed line indicates the expected limit in the absence of a signal. 
Predictions for non-zero values of the anomalous couplings are also shown. 
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Figure 4: The 95% CL upper bound on the anomalous couphng ^"^ as a function of the Higgs 
mass, as obtained from the results of the search for the Standard Model Higgs boson [8]. The 
dashed line indicates the expected limit in the absence of a signal. The dark and light shaded 
bands around the expected line correspond to the 68.3% and 95.4% probability bands, denoted 
by la and 2a respectively. 
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Figure 5: The theoretical predictions for the ratios R = {a^'-^ x Br )/(o"^^ x Br ) for the 
e+e^ — > HZ channel for the couplings a) d,h) ds, c) Ag'^ and d) Ak^. The solid line corresponds 
to the decay H — > ff and the dashed line to H — > 77. The predictions refer to mn = 100 GeV. 
The ratios for the two decay modes coincide for Agf and Akj. 
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Figure 6: Regions excluded at 95% CL as a function of the Higgs mass for the anomalous 
couplings: a) d, b) rf^, c) Agf and d) Ak^. The limits on each coupling are obtained under 
the assumption that the other three couplings are equal to zero. The dashed line indicates 
the expected limit in the absence of a signal. The different hatched regions show the limits 
obtained by the most sensitive analyses: e+e~ —>■ H7 — > 777, e+e~ — e"'"e~H — e"'"e~77, 
e+e" H7 Z77, e+e" HZ and e+e" ^ H7 ^ WW(*)7. 
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Figure 7: Regions excluded at 95% CL for: a) the partial widths r(H ^ Z7) vs. r(H ^ 77) 
and b) the branching fractions Br(H Z7) vs. Br(H 77) in presence of the d and ds 
anomalous couplings. Two values of the Higgs boson mass are considered. The results are 
consistent with the tree level Standard Model expectations r(H — > Z7) f» r(H — 77) 0. 
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